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Measuring Engagement Impact on 
Communities: Challenges and Opportunities 

Mary Hutchinson 


Abstract 

This article describes the author’s reflections on a service¬ 
learning course at Penn State Lehigh Valley, a campus of The 
Pennsylvania State University. The author provides background 
about the university, the community need, and the service¬ 
learning course. Reflections from assessing two semesters of the 
service-learning course are provided. 

Penn State Lehigh Valley 

, enn State Lehigh Valley is a branch campus of the 
Pennsylvania State University with approximately 900 
students located in the third-largest metropolitan area 
of Pennsylvania (after Philadelphia and Pittsburgh). The author, 
a faculty member in English as second language education who 
studies sociocultural diversity and civic engagement, offered a 
service-learning activity as part of a required applied linguistics 
(teaching English grammar) course focused on preparing pre¬ 
service teachers to work with English language learners. The pre¬ 
service teachers were enrolled in a baccalaureate degree program 
in elementary education, which provides graduates with K-6 certi¬ 
fication, and the option for an additional endorsement in teaching 
English as a second language. 

The Community Need 

In Pennsylvania, a state-funded literacy tutoring program 
was created to establish partnerships between community-based 
literacy providers and universities “for the purpose of engaging 
college students in local efforts to help overcome the illiteracy 
problem” (Sherow, 2000). The programs overall goals are to “mobi¬ 
lize, train and retain college students as adult literacy volunteers, 
promote and support the volunteer engagement of college students 
... [and to] provide adult learners with ... instruction and support 
needed to attain their goals” (Pennsylvania Department of Education, 
2011 ). 

The need for well-trained adult literacy tutors is enormous, 
ongoing, and increasing. A report by ProLiteracy America (2003) 
found that “while the number of adults seeking help grows year by 
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year, government funding for literacy programs remains low” (p. 3). 

Therefore, the literacy tutoring 
program provided a vital ser¬ 
vice in linking higher education 
institutions with adult literacy 
programs in order to address this 
need. 

The author, in need of a 
community-based partner for 
her service-learning course, con¬ 
tacted the literacy tutoring pro¬ 
gram and was referred to a local 
agency that served the Lehigh Valley area. The agency is a feder¬ 
ally- and state-funded organization that provides a variety of free or 
low-cost adult education literacy programs. Its overall goal related 
to English as a second language instruction is to help learners 
improve their English reading, writing, speaking, and listening 
skills for the workplace and for everyday life. Many of the adults 
enrolled in the program are studying for the American citizenship 
test as well, and the curriculum reflects this focus. In addition, as 
part of the program, the English language learners establish indi¬ 
vidual learning goals based on their personal needs, which may 
include obtaining a General Education Diploma (GED) and/or 
gaining workforce skills. 

The Service-Learning Activity 

The overall goals of Penn State Lehigh Valley’s applied linguis¬ 
tics course are to introduce pre-service teachers to current theo¬ 
retical issues related to pedagogical grammars and to provide them 
with an opportunity to apply their developing skills of linguistic 
analysis to recognize, analyze, and remediate grammatical errors 
by working directly with English language learners. An additional 
goal is to encourage the students to develop an understanding of 
and appreciation for working with learners whose cultural back¬ 
ground and experiences are often vastly different from their own. 
Exposing pre-service teachers to opportunities for engaging with 
diverse learners is important, particularly since “new teachers and 
teachers in the process of receiving their credential through intern 
or residency programs are placed disproportionately in schools 
and classrooms with large numbers of English language learners” 
(Working Group on ELL Policy, 2009, p. 12). 


“The need for 
well-trained adult 
literacy tutors is 
enormous, ongoing, 
and increasing” 
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To meet these goals, the author developed a service-learning 
component for her course. Service-learning can be defined as “a 
form of experiential education in which students engage in activi¬ 
ties that address human and community needs together with struc¬ 
tured opportunities intentionally designed to promote student 
learning and development” (Jacoby & Associates, 1996, p. 5). “Properly 
designed service-learning courses relate the community service 
experience to the course material and require that students reflect 
on their experiences” (Sax &Astin, 1997, p. 25). 

An initial meeting between Penn State Lehigh Valley and the 
adult literacy agency identified a common purpose and aim, and 
the English Language Learners Literacy Project partnership was 
formed to meet a number of related goals. 

The English Language Learners Literacy Project has two over¬ 
arching goals: 

• addressing the language and literacy needs of adult 
learners in the community whose primary language 
is not English; and 

• assisting these learners in attaining their personal 
learning goals as family members, workers, and 
citizens. 

Higher education goals include providing pre-service teachers 
with opportunities for two types of accomplishment: 

• applying their developing skills of linguistic analysis 
by tutoring English language learners; and 

• developing an understanding and appreciation for 
working with culturally and linguistically diverse 
learners. 

The adult literacy agency is focused on two main goals: 

• providing supplemental tutoring services to English 
language learners; and 

• working in collaboration with higher education to 
address local community literacy needs. 

About the Service-Learning Activity 

The English Language Learners Literacy Project was imple¬ 
mented with a cohort of 22 pre-service teachers who served 63 
English language learners by providing 1,090 hours of tutoring ser¬ 
vices. The students spent 3 to 4 hours each week over the course of 
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a semester providing tutoring in math, reading, writing, and com¬ 
munication skills to the English language learners. Approximately 
104 English language learners participated in the students’ tutoring 
during the two semesters. Only 63 English language learners, how¬ 
ever, were continuously enrolled in the program during this time. 

The adult learners were asked to establish personal learning 
goals at the beginning of the program, such as acquiring a GED, 
obtaining U.S. citizenship, registering to vote, leaving public assis¬ 
tance, or improving literacy skills in order to find employment. 
The students worked with the learners on these personal goals 
and informally kept track of the learners’ progress in their tutor 
logs, which were shared with the agency for reporting purposes. In 
addition, the agency tracked learner progress through the National 
Reporting System for Adult Education, which began in the 1990s 
in an effort to provide a mechanism for agencies to show program 
accountability and effectiveness by “collecting data on adult educa¬ 
tion student outcomes” (U.S. Department of Education, n.d.). 

Measuring the Impact of the 
Service-Learning Activity 

In order to trace the impact of the English Language Learners 
Literacy Project, an assessment was undertaken to explore three 
research questions: 

1. What impact does the project have on pre-service 
teacher understanding and knowledge of working 
with English language learners? 

2. What impact does the project have on the adult lit¬ 
eracy agency and its services? 

3. What impact does the project have on English lan¬ 
guage learners and their progress? 

The data were collected by the author, the director of the lit¬ 
eracy provider program, and the program coordinator for the adult 
literacy agency. IRB approval was secured by the researcher to 
examine the impact of service-learning on the pre-service teachers. 

The Sample of Students Assessed 

The 22 assessment participants represent two consecutive 
cohort sections over a period of one academic year. The 22 pre¬ 
service teacher students fit the profile of the “typical teacher candi¬ 
date” (Lowenstein, 2009, p. 166). They were primarily female (n = 17; 
77%), White (n = 18; 86%, 21 years of age or younger (n = 13; 59%), 
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and monolingual (n = 20; 91%). The majority of them had limited 
exposure to working with English language learners. 

Data Collection Methods 

Data for the assessment was gathered from a variety of sources, 
including reflective writings, a post-course community-based 
learning survey, a college student questionnaire, and data from 
the adult literacy agency. Details about these data sources and the 
analysis are provided in the sections below. 

• Reflective writings. The pre-service students reflected 
on their service-learning experience in weekly tutor 
logs. This data provided an opportunity to glean an 
understanding of their developing awareness of the 
community and the needs of the learners. Although 
relying on student self-perceptions can be prob¬ 
lematic, Matthews and Zimmerman (1999) found 
“that qualitative methods were best for determining 
whether students developed particular benefits of ser¬ 
vice-learning” (p. 386). 

• Post-course community-based learning survey. The 

students were asked to respond to questions from the 
Community-Based Learning Student Survey (Gelmon, 
Holland, Driscoll, Spring, & Kerrigan, 2001) related to 
their attitude toward and experience with commu¬ 
nity involvement. This instrument was designed 
“to describe students’ perspectives and attitudes on 
issues related to their experience in a service-learning 
course” (p. 30). 

• College student questionnaire. At the end of the 
tutoring experience, the pre-service teachers com¬ 
pleted a college student questionnaire for the literacy 
tutoring program. In addition to basic demographic 
information, the students responded to questions 
about two specific aspects of the service-learning 
experience: the accomplishments they felt as tutors, 
and the challenges they encountered. 

• Adult literacy agency data. The agency kept formal 
and informal records on the adult learner partici¬ 
pants’ progress, and the types of assistance the pre¬ 
service teachers provided. Formally, the learners 
were given pre- and posttests of language proficiency 
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developed by the National Reporting System for 
Adult Education (U.S. Department of Education, n.d.). 
Informally, the learners themselves set personal goals 
for their learning when they entered the adult literacy 
agency and reviewed these annually as they progressed 
through the program. 

Data Analysis 

Qualitative analysis was employed to examine the pre-service 
teachers’ reflective writings. Specifically, the reflections were coded 
using open, descriptive coding (through HyperResearch data anal¬ 
ysis software), which allowed the author to identify statements 
related to the students’ knowledge and understanding of working 
with English language learners as expressed in their tutor logs. 

The post-course assessment instruments (Community-Based 
Learning Student Survey, Gelmon et al, 2001, and the college student 
questionnaire) were used to determine the impact of service¬ 
learning on the pre-service teachers. The Community-Based 
Learning Student Survey provides a series of questions designed on 
a 5-point Likert scale of Strongly Agree, Agree, Neutral, Disagree, 
and Strongly Disagree. At the end of the applied linguistics course, 
the students completed this survey to provide feedback about ser¬ 
vice-learning and how it influenced their knowledge of and attitude 
toward working with the community. The college student ques¬ 
tionnaire was a short, open-ended online survey that the students 
completed at the end of the program to provide data about whom 
they worked with during the time frame of the service, what they 
felt they accomplished, and what challenges they encountered. The 
director of the literacy provider program compiled this feedback. 

The test data from the learners’ pre- and post-program lan¬ 
guage proficiency assessment (National Reporting System for Adult 
Education, U.S. Department of Education, n.d.) was compiled by the 
program coordinator of the adult literacy agency (These data are 
often reported back to funding sources as a measure of adult learner 
progress in a program). The coordinator also collected informa¬ 
tion about the learners’ personal learning goals and their progress 
related to these goals, as well as anecdotal data about the impact of 
the tutoring program. 

Findings from the Assessment 

The purpose of this assessment was to examine the impact 
of the English Language Learners Literacy Project on pre-service 
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teachers and their knowledge and understanding of working with 
culturally and linguistically diverse learners, as well as on the com¬ 
munity, including the adult literacy agency and its services and the 
clients who enroll in its programs. The data collected to measure 
this impact were gathered from four distinct data sources: pre-ser¬ 
vice teachers’ reflective writings, a post-course community-based 
learning survey, a college student questionnaire, and adult literacy 
agency learner data. The findings from each of these sources are 
discussed in detail in the following subsections. 

Reflective writings. The early tutor logs reflected the pre-ser¬ 
vice teachers’ initial overall concerns about working with English 
language learners. The words intimidating, inadequate, and nervous 
permeated their reflections; many of them transferred these feel¬ 
ings to the learners themselves: 

I tried very hard to walk in that first day with high 
expectations and wonderful notions of how much these 
students were going to learn; but I have to admit that I 
wasn’t expecting much from them or from me. 

As time went on, these feelings dissipated and were replaced with 
appreciation for the English language learners. Most of the students 
were shocked to learn about the backgrounds and experiences of 
these learners: 

Working with [this learner] opened my eyes to the level 
of commitment many English language learners have 
for this program. He was often required to work 12 hour 
shifts and yet he found time to attend English classes 
and tutoring sessions several times a week. 

Many of the pre-service teachers used the word “respect” to 
describe their change in attitude and newfound admiration for 
their own country. As one student stated, “It was a humbling expe¬ 
rience to learn of the struggles many English language learners deal 
with on a regular basis and how much they appreciate the oppor¬ 
tunities afforded to them in America.” 

A frustration that would surface often was the lack of con¬ 
sistent attendance among the learners. Some English language 
learners were periodically absent or stopped attending altogether. 
The pre-service teachers who were able to work with their learners 
on a consistent basis were able to see growth and commented on 
this progress in their tutor logs, but this was a rarity. 
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By the end of the course, many students mentioned the posi¬ 
tive impact that the experience had on them overall. Many of them 
alluded to how much they learned about themselves as people 
living in a multicultural/multilingual world. 

[The learners] came from all corners of the globe 
. . . South and Central America . . . Brazil, Mexico, 
Dominican Republic, and Guatemala . . . Syria, Egypt, 
and China. These students brought beautiful and dif¬ 
fering perspectives and experiences to the class. They 
each respected each other and the tutors and were 
incredible ambassadors to their native countries. 

Perhaps the most telling change came from one student who was 
resistant to the service-learning activity at the beginning: “In the 
end, I have walked away from this experience with a more open- 
minded attitude.” 

Post-course community-based learning survey. The students 
were asked to respond to questions from the Community-Based 
Learning Student Survey (Gelmon et al, 2001) related to their atti¬ 
tude toward and experience with community involvement. When 
asked whether they felt the community participation aspect of this 
course showed them how to become more involved in the com¬ 
munity, 14 of the 22 felt that it had. All but 2 students felt that the 
community work they did benefited the community; 18 of the 22 
felt that the work helped them to become more aware of the needs 
of their community. 

In addition, the students were asked about their attitude toward 
working with people from different cultural backgrounds. All of 
them agreed or strongly agreed that they felt comfortable working 
with cultures other than their own, and 12 of the 22 indicated that 
the community work made them aware of some of their own biases 
and prejudices. 

College student questionnaire. At the end of the tutoring 
experience, the pre-service teachers completed a college student 
questionnaire for the literacy tutoring program. In addition to 
basic demographic information, the students responded to two 
questions addressing specific aspects of the service-learning expe¬ 
rience: the accomplishments they felt as tutors, and the challenges 
they encountered. 

Many of the students identified a strong sense of accomplish¬ 
ment in understanding what it is like to work with diverse learners 
who have different needs. They pointed to specific instances when 
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they helped the English language learners with topics such as 
“basic sight words and reading,” 
or “understanding] challenges 
with conversational English.” 

However, they also expressed 
frustration about the challenge 
of working with English lan¬ 
guage learners. Many alluded 
to the “language barriers,” and 
how these perceived obstacles 
had had an impact on their 
ability to explain the content in 
a way that was comprehensible. 

Nevertheless, they tried to find 
different ways to help English language learners understand the 
material, and do so “on a level for adults to understand without 
making them feel inferior.” 

Adult literacy agency data. As the English Language Learners 
Literacy Project unfolded, collecting data about the impact of the 
project on the community became a significant challenge. It was 
difficult to obtain direct correspondence between the tutoring 
provided and the progress of a particular learner because of the 
many layers of intervention provided by the agency. In addition, 
the lack of retention and persistence of many of the English lan¬ 
guage learners required that the pre-service teachers work with 
more than one learner. Still, the agency was able to provide general 
feedback about the learners’ progress, and about the impact of the 
tutors on their overall program. 

The agency reported that 63 English language learners were 
served consistently by the 22 pre-service teachers who provided 
approximately 1,090 hours of tutoring. These represented tutor 
hours that the agency would not have provided without the English 
Language Learners Literacy Project. Of the 63 English language 
learners, more than one third (22) achieved one or more of their 
personal goals: 4 had either gotten, kept, or advanced in a job; 1 
was able to help children with schoolwork; 1 was able to increase 
involvement in the community; 13 had attained a better under¬ 
standing of citizenship skills; and 2 had received their U.S. citizen¬ 
ship. It must be kept in mind, however, that these learners had been 
a part of the adult literacy agency program prior to this assessment. 

In addition to individual goal setting, the learners were also 
tested through the National Reporting System for Adult Education 
(U.S. Department of Education, n.d.). Of the 63 English language 


“Many of the students 
identified a strong sense 
of accomplishment in 
understanding what 
it is like to work with 
diverse learners who 
have different needs.” 
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learners, approximately 67% (n = 42) who started on a low to 
intermediate English as a second language level moved up at least 
one level based on the National Reporting System scale during the 
time period of this project. This figure exceeded the literacy tutor 
program standards for adult learner achievement. 

Anecdotally, the adult literacy agency coordinator conveyed 
that the students provided a vital service to the program. Many of 
the learners expressed appreciation for the support they received 
in helping them achieve their goals. 

Discussion 

The pre-service teachers entered the applied linguistics course 
with little experience working with English language learners, but 
left with a clearer understanding of, or even an appreciation for, 
the struggles and needs of these learners. By the end of the course, 
all of them felt that they could indeed work with others whose 
background and experiences were different from their own. This 
is an important realization, because “teachers must be willing to 
learn not only who their students are but also who they, themselves, 
are as cultural beings and how that strongly colors their teaching” 
(Pransky & Bailey, 2002/2003, p. 371). 

The influence of the English Language Learners Literacy 
Project on the agency was evident in the number of tutor hours 
provided. It was clear that all of the learners were given opportu¬ 
nities for tutoring support that the agency might not have been 
able to provide. This relationship among the college students, the 
literacy tutor program, and the adult literacy agency is an impor¬ 
tant one, as it provides English language learners with the time 
and attention they would not have had in a larger classroom set¬ 
ting. This confirms earlier research that found that service-learning 
projects often fill a need in community-based agencies with limited 
staff and resources (Eyler, Giles, Stenson, & Gray, 2001). 

Sustaining the University-Community 
Partnership 

It is clear from the assessment data reported here that the 
applied linguistics service-learning course filled a need in the 
community by providing individual and small group tutoring for 
English language learners enrolled in a local adult literacy agency 
program. Conversely, the university-community partnership pro¬ 
vided pre-service teachers with the opportunity to learn firsthand 
the skills and strategies needed to teach English and to develop an 
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understanding of and appreciation for working with culturally and 
linguistically diverse learners. This mutually beneficial relation¬ 
ship is important for both partners, and is projected to continue 
as long as the pre-service teacher program exists and the adult lit¬ 
eracy agency needs tutors. Because the agency relies on a variety of 
funding sources to support its programs, it most likely will be able 
to sustain this service-learning project for years to come. 

Limitations of the Assessment 

Although it appears that the pre-service teachers were able 
to understand and connect with these learners, more data are 
needed to determine the significance beyond their reflective writ¬ 
ings. Employing the Community-Based Learning Student Survey 
(Gelmon et al„ 2001 J in a pre- and posttest format (or another instru¬ 
ment such as the Civic Attitudes and Skills Questionnaire, Moely, 
McFarland, Miron, Mercer, & llustre, 2002) might provide further evi¬ 
dence about the pre-service teachers’ growth in understanding and 
change in attitude. 

Measuring the impact of the English Language Learners 
Literacy Project on the learners themselves was difficult. Although 
many of the English language learners were able to achieve some 
of their personal learning goals, it was not clear how much direct 
influence the pre-service teachers had in these accomplishments 
because of the multifaceted intervention approach employed by 
the agency and because of the length of time the learners spent in 
their programs. This is the challenge of measuring impact on com¬ 
munities alluded to by Gelmon et al. (2001). Those interested in 
understanding this impact need to determine “what is reasonable 
to expect and accomplish with the service-learning activity,” and 
through the assessment process come to understand “the barriers 
and facilitators of these accomplishments” (p. 84). The process is 
complicated in initiatives such as the English Language Learners 
Literacy Project where service-learning is not the only intervention. 

Plans for Future Research 

The data from this initial inquiry focuses attention on the need 
to measure the impact on the community, a neglected aspect of 
service-learning research (Cruz & Giles, 2000; Gazley & Littlepage, 
2006; Giles, 2010). The findings help to shed light on the challenges 
inherent in measuring the direct impact of service-learning on 
community members, particularly in situations in which multiple 
interventions exist. Under these circumstances, Cruz and Giles 
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(2000) propose “that the university-community partnership itself 
be the unit of analysis” (p. 31), and that the impact on the com¬ 
munity be measured in those terms. In the case of the English 
Language Learners Literacy Project, future research could focus 
on several aspects of the project. 

First, the English language learners could be divided into 
experimental (those who receive tutoring) and control groups 
(those who do not), since the tutoring is supplemental to the 
agency programs and the learners self-select to engage with the 
tutoring support. Data about the English language learners’ pre- 
and postprogram English language proficiency levels (as measured 
by the National Report System scales or other instruments) could 
be used to compare progress of both groups within the specified 
period of the project. 

Second, longitudinal data about the project could be exam¬ 
ined since most English language learners spend multiple years 
in agency programs. A variety of questions merit exploration. 
Do learners who participate in tutoring achieve their goals ear¬ 
lier than those who do not receive these services? Do they score 
greater gains on the tests that measure adult learner achievement? 
Tracking these progress indices could, over time, provide a measure 
of the overall effect of tutoring by pre-service teachers on English 
language learners. 

Finally, the English Language Learners Literacy Project part¬ 
ners need to perform further research not only to determine what 
instruments could be used to measure impact, but also to analyze 
the data collected. Service-learning in higher education is often a 
short-term initiative for students, but this does not preclude higher 
education institutions from working with community-based agen¬ 
cies over the long term to determine the effects of these projects. 
As Sandy and Holland (2006) found in their assessment of com¬ 
munity partnerships, “a growing openness to learn more about the 
perspectives of community members and a willingness to trans¬ 
form our practice in light of their input” (p. 31) has the potential to 
improve service-learning practice. 

Conclusion 

The findings from this assessment highlight the positive impact 
that the English Language Learners Literacy Project partnership 
had on both the students participating in the service-learning 
activity, and the community members in the program. They also 
underscore the need to systematically gather information about 
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impact beyond the methods used in this assessment. Still, those 
interested in implementing a similar project will have a foundation 
on which to develop a clear agenda for organizing a similar service¬ 
learning activity, for implementing effective data collection strate¬ 
gies, and for analyzing the data to assess overall program impact as 
well as the impact on individual participants. 
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